Conversation
This reverts commit e098683.
* origin/master: remote files
DavidDiezIb
approved these changes
Sep 21, 2023
Member
Author
|
So even though the implementation at the time of writing this comment seems to work okay: event, analysis trees are correctly duplicated and metadata are merged, there are some wierd issues when doing the merge where it fails on large number of events after processing many events. Not sure why but I will not merge this until (if) the problem is addressed... |
Member
|
I think doing the merging of analysis tree data is in the scope of |
Member
Author
The original idea of this PR was to have some process that would also merge the metadata. If you just need to merge the analysis tree you can just instantiate a RDataFrame with multiple files as source and do an snapshot. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This updates the file merging logic so that metadata are also merged.
It is assumed that this is only used to merge files that are generated from an identical configuration file (up to run number, random seed, etc.). The first file metadata will be propagated to the merge file, except for
TRestGeant4Metadatathat will be the product of the merging of other metadatas (add up number of primaries, etc.).Metadata, geometry, event and analysis tree will be present in the final file (product of merge).
Originally this merge macro modified the event id of the events if it was duplicated across the new merge file. This is no longer easy to do since we had to use
TFileMergerto merge the trees (could not find another way), so there will be duplicated event ids.